{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ur8xi4C7S06n"
   },
   "source": [
    "# Running a Hyperparameter Tuning Job with Vertex Training\n",
    "\n",
    "## Learning objectives\n",
    "\n",
    "In this notebook, you learn how to:\n",
    "\n",
    "1. Create a Vertex AI custom job for training a model. \n",
    "2. Launch hyperparameter tuning job with the Python SDK.\n",
    "3. Cleanup resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook demonstrates how to run a hyperparameter tuning job with Vertex Training to discover optimal hyperparameter values for an ML model. To speed up the training process, `MirroredStrategy` from the `tf.distribute` module is used to distribute training across multiple GPUs on a single machine.\n",
    "\n",
    "In this notebook, you create a custom-trained model from a Python script in a Docker container. You learn how to modify training application code for hyperparameter tuning and submit a Vertex Training hyperparameter tuning job with the Python SDK.\n",
    "\n",
    "### Dataset\n",
    "\n",
    "The dataset used for this tutorial is the [horses or humans dataset](https://www.tensorflow.org/datasets/catalog/horses_or_humans) from [TensorFlow Datasets](https://www.tensorflow.org/datasets). The trained model predicts if an image is of a horse or a human.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/distributed-hyperparameter-tuning.ipynb) -- try to complete that notebook first before reviewing this solution notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "### Install additional packages\n",
    "\n",
    "Install the latest version of Vertex SDK for Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "wyy5Lbnzg5fi"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-cloud-aiplatform in /opt/conda/lib/python3.7/site-packages (1.13.0)\n",
      "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.3.0)\n",
      "Requirement already satisfied: proto-plus>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.20.3)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (21.3)\n",
      "Requirement already satisfied: google-cloud-bigquery<3.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.34.3)\n",
      "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.4.1)\n",
      "Requirement already satisfied: google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.7.1)\n",
      "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.35.0)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.56.1)\n",
      "Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (3.20.1)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (2.27.1)\n",
      "Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.46.1)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.46.1)\n",
      "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.8.2)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.3.2)\n",
      "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.3.0)\n",
      "Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-resource-manager<3.0.0dev,>=1.3.3->google-cloud-aiplatform) (0.12.4)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=14.3->google-cloud-aiplatform) (3.0.9)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (59.8.0)\n",
      "Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.15.0)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (0.2.7)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (4.2.4)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (4.8)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.1.2)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (2021.10.8)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (3.3)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.26.9)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (2.0.12)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.15.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (0.4.8)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.21)\n"
     ]
    }
   ],
   "source": [
    "# Install necessary dependencies\n",
    "! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhq5zEbGg0XX"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "EzrelQZ22IZj"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs\n",
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BF1j6f9HApxa"
   },
   "source": [
    "### Set up your Google Cloud project\n",
    "\n",
    "1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n",
    "\n",
    "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
    "Cloud SDK uses the right project for all the commands in this notebook.\n",
    "\n",
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WReHDGG5g0XY"
   },
   "source": [
    "#### Set your project ID\n",
    "\n",
    "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "oM1iC_MfAts1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID:  qwiklabs-gcp-00-b9e7121a76ba\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "PROJECT_ID = \"qwiklabs-gcp-00-b9e7121a76ba\"  # Replace your Project ID here \n",
    "\n",
    "# Get your Google Cloud project ID from gcloud\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID: \", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJYoRfYng0XZ"
   },
   "source": [
    "Otherwise, set your project ID here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "riG_qUokg0XZ"
   },
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
    "    PROJECT_ID = \"qwiklabs-gcp-00-b9e7121a76ba\"   # Replace your Project ID here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7325b6915276"
   },
   "source": [
    "Set project ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "Tu9EgBI-WgzD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "06571eb4063b"
   },
   "source": [
    "#### Timestamp\n",
    "\n",
    "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "697568e92bd6"
   },
   "outputs": [],
   "source": [
    "# Import necessary librarary\n",
    "from datetime import datetime\n",
    "\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zgPO1eR3CYjk"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "When you submit a custom training job using the Cloud SDK, you will need to provide a staging bucket.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
    "Cloud Storage buckets.\n",
    "\n",
    "You may also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n",
    "available](https://cloud.google.com/vertex-ai/docs/general/locations#available_regions). You may\n",
    "not use a Multi-Regional Storage bucket for training with Vertex AI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "MzGDU7TWdts_"
   },
   "outputs": [],
   "source": [
    "BUCKET_URI = \"gs://qwiklabs-gcp-00-b9e7121a76ba\"  # Replace your Bucket name here\n",
    "REGION = \"us-central1\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "cf221059d072"
   },
   "outputs": [],
   "source": [
    "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://qwiklabs-gcp-00-b9e7121a76ba\":  # Replace your Bucket name here \n",
    "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "xvrfB6ASU-Jb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://qwiklabs-gcp-00-b9e7121a76baaip-20220526063817\n"
     ]
    }
   ],
   "source": [
    "print(BUCKET_URI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EcIXiGsCePi"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "NIq7R4HZCfIc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://qwiklabs-gcp-00-b9e7121a76baaip-20220526063817/...\n"
     ]
    }
   ],
   "source": [
    "# Create your bucket\n",
    "! gcloud storage buckets create --location $REGION $BUCKET_URI"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ucvCsknMCims"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "vhOb7YnwClBb"
   },
   "outputs": [],
   "source": [
    "# Give access to your Cloud Storage bucket\n",
    "! gcloud storage ls --all-versions --long $BUCKET_URI"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XoEqT2Y4DJmf"
   },
   "source": [
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "pRUOFELefqf1"
   },
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "import os\n",
    "import sys\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud.aiplatform import hyperparameter_tuning as hpt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "reBCSTKOg47l"
   },
   "source": [
    "### Write Dockerfile\n",
    "\n",
    "The first step in containerizing your code is to create a Dockerfile. In the Dockerfile, you'll include all the commands needed to run the image such as installing the necessary libraries and setting up the entry point for the training code.\n",
    "\n",
    "This Dockerfile uses the Deep Learning Container TensorFlow Enterprise 2.5 GPU Docker image. The Deep Learning Containers on Google Cloud come with many common ML and data science frameworks pre-installed. After downloading that image, this Dockerfile installs the [CloudML Hypertune](https://github.com/GoogleCloudPlatform/cloudml-hypertune) library and sets up the entrypoint for the training code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "e231837fe138"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing Dockerfile\n"
     ]
    }
   ],
   "source": [
    "%%writefile Dockerfile\n",
    "\n",
    "FROM gcr.io/deeplearning-platform-release/tf2-gpu.2-5\n",
    "WORKDIR /\n",
    "\n",
    "# Installs hypertune library\n",
    "RUN pip install cloudml-hypertune\n",
    "\n",
    "# Copies the trainer code to the docker image.\n",
    "COPY trainer /trainer\n",
    "\n",
    "# Sets up the entry point to invoke the trainer.\n",
    "ENTRYPOINT [\"python\", \"-m\", \"trainer.task\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4c2ea367c79d"
   },
   "source": [
    "### Create training application code\n",
    "\n",
    "Next, you create a trainer directory with a `task.py` script that contains the code for your training application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "MjJTYC86hPOZ"
   },
   "outputs": [],
   "source": [
    "# Create trainer directory\n",
    "\n",
    "! mkdir trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ea83c1253a74"
   },
   "source": [
    "In the next cell, you write the contents of the training script, `task.py`. This file downloads the _horses or humans_ dataset from TensorFlow datasets and trains a `tf.keras` functional model using `MirroredStrategy` from the `tf.distribute` module.\n",
    "\n",
    "There are a few components that are specific to using the hyperparameter tuning service:\n",
    "\n",
    "* The script imports the `hypertune` library. Note that the Dockerfile included instructions to pip install the hypertune library.\n",
    "* The function `get_args()` defines a command-line argument for each hyperparameter you want to tune. In this example, the hyperparameters that will be tuned are the learning rate, the momentum value in the optimizer, and the number of units in the last hidden layer of the model. The value passed in those arguments is then used to set the corresponding hyperparameter in the code.\n",
    "* At the end of the `main()` function, the hypertune library is used to define the metric to optimize. In this example, the metric that will be optimized is the the validation accuracy. This metric is passed to an instance of `HyperTune`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "9b52fd75d90f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing trainer/task.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile trainer/task.py\n",
    "\n",
    "import argparse\n",
    "import hypertune\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "def get_args():\n",
    "  \"\"\"Parses args. Must include all hyperparameters you want to tune.\"\"\"\n",
    "\n",
    "  parser = argparse.ArgumentParser()\n",
    "  parser.add_argument(\n",
    "      '--learning_rate', required=True, type=float, help='learning rate')\n",
    "  parser.add_argument(\n",
    "      '--momentum', required=True, type=float, help='SGD momentum value')\n",
    "  parser.add_argument(\n",
    "      '--units',\n",
    "      required=True,\n",
    "      type=int,\n",
    "      help='number of units in last hidden layer')\n",
    "  parser.add_argument(\n",
    "      '--epochs',\n",
    "      required=False,\n",
    "      type=int,\n",
    "      default=10,\n",
    "      help='number of training epochs')\n",
    "  args = parser.parse_args()\n",
    "  return args\n",
    "\n",
    "\n",
    "def preprocess_data(image, label):\n",
    "  \"\"\"Resizes and scales images.\"\"\"\n",
    "\n",
    "  image = tf.image.resize(image, (150, 150))\n",
    "  return tf.cast(image, tf.float32) / 255., label\n",
    "\n",
    "\n",
    "def create_dataset(batch_size):\n",
    "  \"\"\"Loads Horses Or Humans dataset and preprocesses data.\"\"\"\n",
    "\n",
    "  data, info = tfds.load(\n",
    "      name='horses_or_humans', as_supervised=True, with_info=True)\n",
    "\n",
    "  # Create train dataset\n",
    "  train_data = data['train'].map(preprocess_data)\n",
    "  train_data = train_data.shuffle(1000)\n",
    "  train_data = train_data.batch(batch_size)\n",
    "\n",
    "  # Create validation dataset\n",
    "  validation_data = data['test'].map(preprocess_data)\n",
    "  validation_data = validation_data.batch(64)\n",
    "\n",
    "  return train_data, validation_data\n",
    "\n",
    "\n",
    "def create_model(units, learning_rate, momentum):\n",
    "  \"\"\"Defines and compiles model.\"\"\"\n",
    "\n",
    "  inputs = tf.keras.Input(shape=(150, 150, 3))\n",
    "  x = tf.keras.layers.Conv2D(16, (3, 3), activation='relu')(inputs)\n",
    "  x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
    "  x = tf.keras.layers.Conv2D(32, (3, 3), activation='relu')(x)\n",
    "  x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
    "  x = tf.keras.layers.Conv2D(64, (3, 3), activation='relu')(x)\n",
    "  x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
    "  x = tf.keras.layers.Flatten()(x)\n",
    "  x = tf.keras.layers.Dense(units, activation='relu')(x)\n",
    "  outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n",
    "  model = tf.keras.Model(inputs, outputs)\n",
    "  model.compile(\n",
    "      loss='binary_crossentropy',\n",
    "      optimizer=tf.keras.optimizers.SGD(\n",
    "          learning_rate=learning_rate, momentum=momentum),\n",
    "      metrics=['accuracy'])\n",
    "  return model\n",
    "\n",
    "\n",
    "def main():\n",
    "  args = get_args()\n",
    "\n",
    "  # Create Strategy\n",
    "  strategy = tf.distribute.MirroredStrategy()\n",
    "\n",
    "  # Scale batch size\n",
    "  GLOBAL_BATCH_SIZE = 64 * strategy.num_replicas_in_sync  \n",
    "  train_data, validation_data = create_dataset(GLOBAL_BATCH_SIZE)\n",
    "\n",
    "  # Wrap model variables within scope\n",
    "  with strategy.scope():\n",
    "    model = create_model(args.units, args.learning_rate, args.momentum)\n",
    "\n",
    "  # Train model\n",
    "  history = model.fit(\n",
    "      train_data, epochs=args.epochs, validation_data=validation_data)\n",
    "\n",
    "  # Define Metric\n",
    "  hp_metric = history.history['val_accuracy'][-1]\n",
    "\n",
    "  hpt = hypertune.HyperTune()\n",
    "  hpt.report_hyperparameter_tuning_metric(\n",
    "      hyperparameter_metric_tag='accuracy',\n",
    "      metric_value=hp_metric,\n",
    "      global_step=args.epochs)\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "  main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2dc0a526f77e"
   },
   "source": [
    "### Build the Container\n",
    "\n",
    "In the next cells, you build the container and push it to Google Container Registry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "a42d0b918ab4"
   },
   "outputs": [],
   "source": [
    "# Set the IMAGE_URI\n",
    "IMAGE_URI = f\"gcr.io/{PROJECT_ID}/horse-human:hypertune\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "360a5271fbdb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending build context to Docker daemon  355.3kB\n",
      "Step 1/5 : FROM gcr.io/deeplearning-platform-release/tf2-gpu.2-5\n",
      "latest: Pulling from deeplearning-platform-release/tf2-gpu.2-5\n",
      "\n",
      "\u001b[1B05cd42bd: Pulling fs layer \n",
      "\u001b[1B353a95ec: Pulling fs layer \n",
      "\u001b[1B996407de: Pulling fs layer \n",
      "\u001b[1B6fc70f16: Pulling fs layer \n",
      "\u001b[1B6be11512: Pulling fs layer \n",
      "\u001b[1B5a9c78ee: Pulling fs layer \n",
      "\u001b[1B690abd59: Pulling fs layer \n",
      "\u001b[1Bdd7682bc: Pulling fs layer \n",
      "\u001b[1B7660f71d: Pulling fs layer \n",
      "\u001b[1B792fb622: Pulling fs layer \n",
      "\u001b[1B43275aca: Pulling fs layer \n",
      "\u001b[1B6f115483: Pulling fs layer \n",
      "\u001b[1B22b52524: Pulling fs layer \n",
      "\u001b[1B9a9f592b: Pulling fs layer \n",
      "\u001b[1B7d1c07e4: Pulling fs layer \n",
      "\u001b[1Bd5e9923d: Pulling fs layer \n",
      "\u001b[13Bbe11512: Waiting fs layer \n",
      "\u001b[1B79b748d5: Pulling fs layer \n",
      "\u001b[1B8bd6f7fb: Pulling fs layer \n",
      "\u001b[1B16bef138: Pulling fs layer \n",
      "\u001b[1B27033824: Pulling fs layer \n",
      "\u001b[1B8679f003: Pulling fs layer \n",
      "\u001b[1B22e7ff04: Pulling fs layer \n",
      "\u001b[1B48a3aec1: Pulling fs layer \n",
      "\u001b[1B637de50b: Pulling fs layer \n",
      "\u001b[1Bf1eb7f16: Pulling fs layer \n",
      "\u001b[1Bf4265c8c: Pulling fs layer \n",
      "\u001b[1B1c49222a: Pulling fs layer \n",
      "\u001b[1Beb38f366: Pulling fs layer \n",
      "\u001b[1Be5017b4d: Pull complete 7.2kB/787.2kBB\u001b[28A\u001b[2K\u001b[30A\u001b[2K\u001b[27A\u001b[2K\u001b[30A\u001b[2K\u001b[25A\u001b[2K\u001b[24A\u001b[2K\u001b[30A\u001b[2K\u001b[22A\u001b[2K\u001b[25A\u001b[2K\u001b[30A\u001b[2K\u001b[21A\u001b[2K\u001b[30A\u001b[2K\u001b[23A\u001b[2K\u001b[30A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[30A\u001b[2K\u001b[30A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[30A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[30A\u001b[2K\u001b[25A\u001b[2K\u001b[30A\u001b[2K\u001b[23A\u001b[2K\u001b[30A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[30A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[29A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[29A\u001b[2K\u001b[25A\u001b[2K\u001b[29A\u001b[2K\u001b[25A\u001b[2K\u001b[29A\u001b[2K\u001b[25A\u001b[2K\u001b[29A\u001b[2K\u001b[25A\u001b[2K\u001b[29A\u001b[2K\u001b[25A\u001b[2K\u001b[29A\u001b[2K\u001b[25A\u001b[2K\u001b[28A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[28A\u001b[2K\u001b[25A\u001b[2K\u001b[28A\u001b[2K\u001b[21A\u001b[2K\u001b[28A\u001b[2K\u001b[21A\u001b[2K\u001b[28A\u001b[2K\u001b[21A\u001b[2K\u001b[28A\u001b[2K\u001b[28A\u001b[2K\u001b[27A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[27A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[26A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[23A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[23A\u001b[2K\u001b[25A\u001b[2K\u001b[23A\u001b[2K\u001b[25A\u001b[2K\u001b[23A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[21A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[18A\u001b[2K\u001b[19A\u001b[2K\u001b[16A\u001b[2K\u001b[19A\u001b[2K\u001b[17A\u001b[2K\u001b[19A\u001b[2K\u001b[17A\u001b[2K\u001b[15A\u001b[2K\u001b[17A\u001b[2K\u001b[25A\u001b[2K\u001b[14A\u001b[2K\u001b[25A\u001b[2K\u001b[13A\u001b[2K\u001b[25A\u001b[2K\u001b[12A\u001b[2K\u001b[15A\u001b[2K\u001b[11A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[8A\u001b[2K\u001b[15A\u001b[2K\u001b[7A\u001b[2K\u001b[15A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[15A\u001b[2K\u001b[6A\u001b[2K\u001b[4A\u001b[2K\u001b[6A\u001b[2K\u001b[25A\u001b[2K\u001b[6A\u001b[2K\u001b[25A\u001b[2K\u001b[6A\u001b[2K\u001b[4A\u001b[2K\u001b[6A\u001b[2K\u001b[4A\u001b[2K\u001b[6A\u001b[2K\u001b[4A\u001b[2K\u001b[5A\u001b[2K\u001b[4A\u001b[2K\u001b[5A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[6A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[6A\u001b[2K\u001b[3A\u001b[2K\u001b[6A\u001b[2K\u001b[3A\u001b[2K\u001b[6A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[6A\u001b[2K\u001b[25A\u001b[2K\u001b[6A\u001b[2K\u001b[25A\u001b[2K\u001b[6A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[2A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[25A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[5A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[25A\u001b[2K\u001b[24A\u001b[2K\u001b[24A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[23A\u001b[2K\u001b[22A\u001b[2K\u001b[22A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[21A\u001b[2K\u001b[20A\u001b[2K\u001b[20A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[19A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[16A\u001b[2K\u001b[16A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[14A\u001b[2K\u001b[13A\u001b[2K\u001b[12A\u001b[2K\u001b[11A\u001b[2K\u001b[10A\u001b[2K\u001b[10A\u001b[2K\u001b[9A\u001b[2K\u001b[8A\u001b[2K\u001b[8A\u001b[2K\u001b[7A\u001b[2K\u001b[7A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[5A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2KDigest: sha256:fb35899e63342d0f5e603b7a567bcac8a6a8fc0744bec058f7da1acdfad90637\n",
      "Status: Downloaded newer image for gcr.io/deeplearning-platform-release/tf2-gpu.2-5:latest\n",
      " ---> 307b41b1aec7\n",
      "Step 2/5 : WORKDIR /\n",
      " ---> Running in 97b24bcb9eb9\n",
      "Removing intermediate container 97b24bcb9eb9\n",
      " ---> a7aabeeb9e44\n",
      "Step 3/5 : RUN pip install cloudml-hypertune\n",
      " ---> Running in 3c72ea75e315\n",
      "Collecting cloudml-hypertune\n",
      "  Downloading cloudml-hypertune-0.1.0.dev6.tar.gz (3.2 kB)\n",
      "Building wheels for collected packages: cloudml-hypertune\n",
      "  Building wheel for cloudml-hypertune (setup.py): started\n",
      "  Building wheel for cloudml-hypertune (setup.py): finished with status 'done'\n",
      "  Created wheel for cloudml-hypertune: filename=cloudml_hypertune-0.1.0.dev6-py2.py3-none-any.whl size=3988 sha256=d528d0f0d7c73b0ef0eb9fce46775b13ea866aadd8a718a3b402260e132396e3\n",
      "  Stored in directory: /root/.cache/pip/wheels/a7/ff/87/e7bed0c2741fe219b3d6da67c2431d7f7fedb183032e00f81e\n",
      "Successfully built cloudml-hypertune\n",
      "Installing collected packages: cloudml-hypertune\n",
      "Successfully installed cloudml-hypertune-0.1.0.dev6\n",
      "\u001b[91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n",
      "\u001b[0mRemoving intermediate container 3c72ea75e315\n",
      " ---> 7d7b9110606f\n",
      "Step 4/5 : COPY trainer /trainer\n",
      " ---> bb89d82023c1\n",
      "Step 5/5 : ENTRYPOINT [\"python\", \"-m\", \"trainer.task\"]\n",
      " ---> Running in 2fbdbf0d910f\n",
      "Removing intermediate container 2fbdbf0d910f\n",
      " ---> f25779342aef\n",
      "Successfully built f25779342aef\n",
      "Successfully tagged gcr.io/qwiklabs-gcp-00-b9e7121a76ba/horse-human:hypertune\n"
     ]
    }
   ],
   "source": [
    "# Build the docker image\n",
    "! docker build -f Dockerfile -t $IMAGE_URI ./"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "526475da6370"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The push refers to repository [gcr.io/qwiklabs-gcp-00-b9e7121a76ba/horse-human]\n",
      "\n",
      "\u001b[1Bd49ddc5a: Preparing \n",
      "\u001b[1B7c5ace32: Preparing \n",
      "\u001b[1Baa5df10d: Preparing \n",
      "\u001b[1B010939aa: Preparing \n",
      "\u001b[1Bc4ea3a81: Preparing \n",
      "\u001b[1B08c5711b: Preparing \n",
      "\u001b[1Bb564e194: Preparing \n",
      "\u001b[1B6808a3d1: Preparing \n",
      "\u001b[1Bbdf9b557: Preparing \n",
      "\u001b[1Bdbc2b748: Preparing \n",
      "\u001b[1Bb8f29c2e: Preparing \n",
      "\u001b[1B7b2f7486: Preparing \n",
      "\u001b[1B97a3e6e4: Preparing \n",
      "\u001b[1Ba5e8117f: Preparing \n",
      "\u001b[1B8124ed57: Preparing \n",
      "\u001b[1B4704bb3d: Preparing \n",
      "\u001b[1B6ef24b4b: Preparing \n",
      "\u001b[1B113f67c8: Preparing \n",
      "\u001b[1B857a1d48: Preparing \n",
      "\u001b[1B97864c52: Preparing \n",
      "\u001b[1Bbaac3e32: Preparing \n",
      "\u001b[1Ba1af4c10: Preparing \n",
      "\u001b[1Ba468ca49: Preparing \n",
      "\u001b[1B205798d1: Preparing \n",
      "\u001b[1Bcd6d4269: Preparing \n",
      "\u001b[1B55c89c2a: Preparing \n",
      "\u001b[1Bb9034da6: Preparing \n",
      "\u001b[1B4fbfce85: Preparing \n",
      "\u001b[1B9ca3db46: Preparing \n",
      "\u001b[1B1a1930ab: Preparing \n",
      "\u001b[1Bf5a43f1f: Preparing \n",
      "\u001b[2Bf5a43f1f: Mounted from deeplearning-platform-release/tf2-gpu.2-5 \u001b[28A\u001b[2K\u001b[30A\u001b[2K\u001b[27A\u001b[2K\u001b[31A\u001b[2K\u001b[26A\u001b[2K\u001b[24A\u001b[2K\u001b[22A\u001b[2K\u001b[21A\u001b[2K\u001b[19A\u001b[2K\u001b[18A\u001b[2K\u001b[16A\u001b[2K\u001b[15A\u001b[2K\u001b[14A\u001b[2K\u001b[12A\u001b[2K\u001b[13A\u001b[2K\u001b[11A\u001b[2K\u001b[9A\u001b[2K\u001b[6A\u001b[2K\u001b[7A\u001b[2K\u001b[4A\u001b[2K\u001b[3A\u001b[2K\u001b[1A\u001b[2K\u001b[2A\u001b[2Khypertune: digest: sha256:8ed36a4791622a7f845f7a3cbcd4b68e20f760b182d7f3079b33be3e0af572a4 size: 7045\n"
     ]
    }
   ],
   "source": [
    "# Push it to Google Container Registry:\n",
    "! docker push $IMAGE_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aaff6f5be7f6"
   },
   "source": [
    "### Create and run hyperparameter tuning job on Vertex AI\n",
    "\n",
    "Once your container is pushed to Google Container Registry, you use the Vertex SDK to create and run the hyperparameter tuning job.\n",
    "\n",
    "You define the following specifications:\n",
    "* `worker_pool_specs`: Dictionary specifying the machine type and Docker image. This example defines a single node cluster with one `n1-standard-4` machine with two `NVIDIA_TESLA_T4` GPUs.\n",
    "* `parameter_spec`: Dictionary specifying the parameters to optimize. The dictionary key is the string assigned to the command line argument for each hyperparameter in your training application code, and the dictionary value is the parameter specification. The parameter specification includes the type, min/max values, and scale for the hyperparameter.\n",
    "* `metric_spec`: Dictionary specifying the metric to optimize. The dictionary key is the `hyperparameter_metric_tag` that you set in your training application code, and the value is the optimization goal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "aec22792ee84"
   },
   "outputs": [],
   "source": [
    "# Define required specifications\n",
    "worker_pool_specs = [\n",
    "    {\n",
    "        \"machine_spec\": {\n",
    "            \"machine_type\": \"n1-standard-4\",\n",
    "            \"accelerator_type\": \"ACCELERATOR_TYPE_UNSPECIFIED\",\n",
    "            \"accelerator_count\": 0,\n",
    "        },\n",
    "        \"replica_count\": 1,\n",
    "        \"container_spec\": {\"image_uri\": IMAGE_URI},\n",
    "    }\n",
    "]\n",
    "\n",
    "metric_spec = {\"accuracy\": \"maximize\"}\n",
    "\n",
    "parameter_spec = {\n",
    "    \"learning_rate\": hpt.DoubleParameterSpec(min=0.001, max=1, scale=\"log\"),\n",
    "    \"momentum\": hpt.DoubleParameterSpec(min=0, max=1, scale=\"linear\"),\n",
    "    \"units\": hpt.DiscreteParameterSpec(values=[64, 128, 512], scale=None),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ffd01019a764"
   },
   "source": [
    "Create a `CustomJob`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "IEpSVz5XZEjg"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://qwiklabs-gcp-00-b9e7121a76baaip-20220526063817\n"
     ]
    }
   ],
   "source": [
    "print(BUCKET_URI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "f2eed1471a24"
   },
   "outputs": [],
   "source": [
    "# Create a CustomJob\n",
    "\n",
    "JOB_NAME = \"horses-humans-hyperparam-job\" + TIMESTAMP\n",
    "\n",
    "# TODO 1\n",
    "my_custom_job = aiplatform.CustomJob(\n",
    "    display_name=JOB_NAME,\n",
    "    project=PROJECT_ID,\n",
    "    worker_pool_specs=worker_pool_specs,\n",
    "    staging_bucket=BUCKET_URI,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0e5ee7ee5ae4"
   },
   "source": [
    "Then, create and run a `HyperparameterTuningJob`.\n",
    "\n",
    "There are a few arguments to note:\n",
    "\n",
    "* `max_trial_count`: Sets an upper bound on the number of trials the service will run. The recommended practice is to start with a smaller number of trials and get a sense of how impactful your chosen hyperparameters are before scaling up.\n",
    "\n",
    "* `parallel_trial_count`:  If you use parallel trials, the service provisions multiple training processing clusters. The worker pool spec that you specify when creating the job is used for each individual training cluster.  Increasing the number of parallel trials reduces the amount of time the hyperparameter tuning job takes to run; however, it can reduce the effectiveness of the job overall. This is because the default tuning strategy uses results of previous trials to inform the assignment of values in subsequent trials.\n",
    " \n",
    "* `search_algorithm`: The available search algorithms are grid, random, or default (None). The default option applies Bayesian optimization to search the space of possible hyperparameter values and is the recommended algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bb6ca1b103ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating HyperparameterTuningJob\n",
      "HyperparameterTuningJob created. Resource name: projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240\n",
      "To use this HyperparameterTuningJob in another session:\n",
      "hpt_job = aiplatform.HyperparameterTuningJob.get('projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240')\n",
      "View HyperparameterTuningJob:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/1248415738746634240?project=585438674354\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "HyperparameterTuningJob projects/585438674354/locations/us-central1/hyperparameterTuningJobs/1248415738746634240 current state:\n",
      "JobState.JOB_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "# Create and run HyperparameterTuningJob\n",
    "\n",
    "# TODO 2\n",
    "hp_job = aiplatform.HyperparameterTuningJob(\n",
    "    display_name=JOB_NAME,\n",
    "    custom_job=my_custom_job,\n",
    "    metric_spec=metric_spec,\n",
    "    parameter_spec=parameter_spec,\n",
    "    max_trial_count=15,\n",
    "    parallel_trial_count=3,\n",
    "    project=PROJECT_ID,\n",
    "    search_algorithm=None,\n",
    ")\n",
    "\n",
    "hp_job.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9Nmi2KIicB7S"
   },
   "source": [
    "** It will nearly take 50 mintues to complete the job successfully.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "396d86efe829"
   },
   "source": [
    "Click on the generated link in the output to see your run in the Cloud Console. When the job completes, you will see the results of the tuning trials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bee87f15ff24"
   },
   "source": [
    "![console_ui_results](tuning_results.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpV-iwP9qw9c"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Otherwise, you can delete the individual resources you created in this tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "sx_vKniMq9ZX"
   },
   "outputs": [],
   "source": [
    "# Set this to true only if you'd like to delete your bucket\n",
    "# TODO 3\n",
    "delete_bucket = False\n",
    "\n",
    "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
    "    ! gcloud storage rm --recursive $BUCKET_URI"   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "distributed-hyperparameter-tuning.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m92",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m92"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
